WORLD INTELLECTUAL PROPERTY ORGANIZATION 
International Bureau 




PCT 

INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(51) International Patent Classification 6 
G06T 15/70 



Al 



(11) Internationa! Publication Number: WO -99/06962 

(43) International Publication Date: 1 1 February 1999 (1 1 .02.99) 



(21) International Application Number: PCT/US98/ 15702 

(22) International Filing Date: 29 July 1998 (29.07.98) 



(30) Priority Data: 

08/905,436 



4 August 1997 (04.08.97) 



US 



(71) Applicant: PIXAR ANIMATION STUDIOS [US/US]; 1001 

West Cutting Boulevard, Richmond, CA 94804-2028 (US). 

(72) Inventors: DEROSE, Anthony, David; 10 Robinhood Drive, 

San Rafael, CA 94901 (US). KASS, Michael; 2329 Oak 
Street, Berkeley. CA 94708 (US). 

(74) Agents: THOMSON, William, E., Jr. et al.; McCutchen, 
Doyle, Brown & Enersen, Three Embarcadero Center, San 
Francisco, CA 94111 (US). 



(81) Designated States: AL, AM, AT, AU, AZ, BA, BB, BG, BR, 
BY CA, CH, CN, CU, CZ, DE, DK, EE, ES, FI, GB, GE, 
GH, GM, HR, HU, ID, IL, IS, JP, KE, KG, KP, KR, KZ, 
LC, LK, LR, LS, LT, LU, LV, MD, MG, MK, MN, MW, 
MX, NO, NZ, PL, PT, RO, RU, SD, SE, SG, SI, SK, SL, TJ, 
TM, TR, TT, UA, UG, UZ, VN. YU, ZW, ARIPO patent 
(GH, GM, KE, LS, MW, SD, SZ, UG, ZW), Eurasian patent 
(AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), European patent 
(AT, BE, CH, CY, DE, DK, ES, FI. FR, GB, GR, IE, IT, 
LU, MC, NL, PT, SE), OAPI patent (BF, BJ, CF, CG, CI. 
CM, GA, GN, GW, ML, MR, NE, SN, TD, TG). 



Published 

With international search report. 



09 

m 



(54) Title: REALISTIC SURFACE SIMULATION IN COMPUTER ANIMATION 




(57) Abstract 

Realistic computer generated character animation is achieved by modeling skin or other covering as a separate, quasi-static surface 
elastically tied to an underlying detailed, articulated model controlled by the animator. The result automatically incorporates realistic 
stretching and sagging, and the combined effect of multiple overlapping animation controls while retaining for the animator frame-by-frame 
control of a detailed model of the character. 



BNSDOCID: <WO 9906962A 1 J_> 



FOR THE PURPOSES OF INFORMATION ONLY 
Codes used to identify States party to the PCX on the front pages of pamphlets publishing international applications under the PCT. 



AL 


Albania 


ES 


AM 


Armenia 


FI 


AT 


Austria 


FR 


AU 


Australia 


GA 


AZ 


Azerbaijan 


GB 


BA 


Bosnia and Herzegovina 


GE 


BB 


Barbados 


GH 


BE 


Belgium 


GN 


BF 


Burkina Faso 


GR 


BG 


Bulgaria 


HU 


BJ 


Benin 


IE 


BR 


Brazil 


IL 


BY 


Belarus 


IS 


CA 


Canada 


IT 


CF 


Central African Republic 


JP 


CC 


Congo 


KE 


CH 


Switzerland 


KG 


CI 


Cdte d* I voire 


KP 


CM 


Cameroon 




CN 


China 


KR 


CU 


Cuba 


KZ 


CZ 


Czech Republic 


LC 


DE 


Germany 


LI 


DK 


Denmark 


LK 


EE 


Estonia 


LR 



Spain 
Finland 
France 
Gabon 

United Kingdom 

Georgia 

Ghana 

Guinea 

Greece 

Hungary 

Ireland 

Israel 

Iceland 

Italy 

Japan 

Kenya 

Kyrgyzstan 

Democratic People's 

Republic of Korea 

Republic of Korea 

Kazakstan 

Saint Lucia 

Liechtenstein 

Sri Lanka 

Liberia 



LS Lesotho 

LT Lithuania 

LU Luxembourg 

LV Latvia 

MC Monaco 

MD Republic of Moldova 

MG Madagascar 

MK The former Yugoslav 
Republic of Macedonia 

ML Mali 

MN Mongolia 

MR Mauritania 

MW Malawi 

MX Mexico 

NE Niger 

NL Netherlands 

NO Norway 

NZ New Zealand 

PL Poland 

PT Portugal 

RO Romania 

RU Russian Federation 
SD Sudan 
SE Sweden 
SG Singapore 



SI 


Slovenia 


SK 


Slovakia 


SN 


Senegal 


sz 


Swaziland 


TD 


Chad 


TG 


Togo 


TJ 


Tajikistan 


TM 


Turkmenistan 


TR 


Turkey 


TT 


Trinidad and Tobago 


UA 


Ukraine 


UG 


Uganda 


US 


United States of A'.r-.- »-:a 


uz 


Uzbekistan 


VN 


Viet Nam 


YU 


Yugoslavia 


zw 


Zimbabwe 



BNSDOCID: <WO 9906962A1_1_> 



PCT/US98/15702 

WO 99/06962 

1 

Realistic Surface Simulation in Computer Animation 
FIELD OF THE INVENTION 

The invention relates generally to the art of computer generated 
animation. More particularly, the invention relates to the realistic modeling of 
5 characters in computer generated animation. 

CROSS-REFERENCE TO RELATED APPLICATIONS 
The following co-pending U.S. Patent applications relate to the present 
application. Each of the listed co-pending applications is assigned to the same 
assignee as the present application. 
10 l. U.S. patent application Serial No. 08/905,434, filed August 4, 

1997 and titled: "TEXTURE MAPPING AND OTHER USES OF SCALAR FIELDS 
ON SUBDIVISION SURFACES IN COMPUTER GRAPHICS AND 
ANIMATION". 

2. U.S. patent application Serial No. 08/905,435, filed August 4, 
15 1 997 and titled: "HYBRID SUBDIVISION IN COMPUTER GRAPHICS". 

BACKGROUND OF THE INVENTION 
To create a three dimensional computer animation, the animator must 
move three dimensional objects and characters about the scene by specifying the 
location of all parts of all objects in every frame, or at least in key frames between 
20 which motion can be interpolated. In the case of realistic animation in which the 
individual characters are highly detailed with many attributes, e.g., possible facial 
expressions, this can be a monumental task. As a first step, the process often begins 
with a detailed physical model of the character which is then scanned and stored in the 
computer as a two dimensional array of control points corresponding to vertices of a 
25 mesh which approximates the model's bounding surface. The control point mesh 

contains sufficient information to recreate the model using either B-Spline patches, a 
polygon mesh, or recursive subdivision surfaces, in sufficient detail to produce a high 

quality rendered image. 

In the traditional "entire kinematic control" approach to computer 
30 generated animation, the animator controls the movement of the character by 

individually specifying the movement of each of the numerous control points. The 



BNSDOCID: <WO 9906962A 1 J_> 



PCT/US98/15702 

WO 99/06962 

2 

difficulty is that a highly detailed human head may be represented by 5000 or more * 
control points. The ability to individually manipulate the locations of that many 
degrees of freedom is both a blessing and a curse. Though it allows the animator to 
direct the motion, facial expressions, and gestures of the characters in great detail, 
5 independently specifying the motion of so many points is an immensely tedious 
process. 

Moreover, because the locations of the control points are independently 
variable and not tied to together, physical constraints, e.g., that they belong to a semi- 
rigid body, must be individually enforced by the animator who must for instance 
10 insure that control points do not crash into each other or otherwise behave in an 

unnatural manner which would give rise to unacceptable artifacts in the final rendered 
scenes. 

One may of course drastically reduce the number of degrees of 
freedom which must be specified and insure that the above described physical 
15 constraints are enforced, by treating all or part of the character as a rigid body thereby 
tying a set of the control points together. One may for instance limit the degrees of 
freedom necessary to specify the movement of a character's jaw by treating the lower 
jaw as a rigid body for which the locations of the control points depend only on their 
initial locations and the angle of rotation of the jaw. 
20 while the constraints of rigid body kinematics greatly lightens the 

modeling load and works well for the animation of actual rigid bodies, e.g., toys and 
robots, and as a first approximation for the large scale movements of extremities, it 
does not result in the realistic animation of people or animals. Animating faces is 
particularly problematic both because of the many small movements which make up 
25 expressive facial gestures and because skin can be both somewhat loose and elastic, so 
that realistic facial movements are neither localized nor representable in terms of 
simple rigid body kinematics, e.g., the skin of a jowly character may move little or not 
at all, much less exactly track the angle of rotation of his jaw. 

One commonly used method for more efficiently specifying the motion 
30 of sets control points which utilizes some of the simplifications of rigid body 
kinematics is the use of animation controls to implement elementary physical 
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movements. In this method, described in more detail below in the discussion of the . 
exemplary embodiment, the motion of numerous points is specified in terms of a 
single control, as in the rigid body case, but the control points affected by the control 
are not required to move the same amount. Rather, their relative motion is determined 
5 by a weight assigned to each point. Though this basically turns a three dimensional 
problem into a one dimensional one, it nonetheless requires the articulation engineer 
to perform the still tedious task of specifying hundreds or thousands of individual 
weights. Moreover, the character's desired motion will often require the 
superposition of multiple elementary simultaneous controls. That is, a single point 
10 may be subject to, i.e., have non-zero weights corresponding to, several different 

simultaneous controls. The ultimate motion of the point is then determined to be the 
sum of the motions required by each control. Unfortunately this result does not 
always mirror reality, e.g., if a character opens its jaw and wiggles its ear at the same 
time, the motion of a point on his cheek is not really the sum of the motions of the 
15 point under those two acts performed independently. 

One ambitious approach to realistically model the behavior of skin and 
avoid the tedious and difficult point by point kinematic approach is to model skin as a 
dynamical system driven by underlying skeletal movements. This approach of 
building a face from the inside out requires solving the equations of motions for the 
20 control points as elements of a semi-elastic damped surface subject to a driving force 
provided by the animator controlled movement of underlying "skeletal" components. 

Even if one could model such a complex system in a way which gave 
realistic results, from the animator's perspective this pure dynamical approach suffers 
from two related shortcomings. The first problem is that truth does not always equal 
25 beauty. Even if one could accurately model the movement of skin resulting from 

underlying rigid body kinematics and the dynamics of the connective tissue, the result 
may not have the appearance desired by the animator. Animators creating computer 
generated animation as artistic works or for commercial entertainment are more 
interested in the appearance of reality than reality and often times not even the 
30 appearance of reality but rather caricature and exaggeration. In any event, animators 
often do not want to give up the degree of control over the final image required by a 
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purely dynamical approach to modeling skin and facial features. Nor do they want to , 
give up control of the details of the character and rely instead on the manipulation of a 
skeleton and musculature and some approximation to the intervening biomechanics. 

Relatedly, a dynamical approach because it involves solving equations 
5 of motion over time, including the effects of momentum, ties the motion of different 
frames together into a single trajectory. An animator may vary the trajectory by 
changing the initial conditions and driving force, but cannot directly alter the 
animation independently on a frame by frame basis. Again this requires that the 
animator relinquish a large chunk of artistic freedom, this time to Newtonian 
10 mechanics. Accordingly, there is a need for a more efficient means of modeling 
characters in computer animation which retains for the animator the ability to 
substantially control the detailed appearance and movement of the character. 

SUMMARY OF THE INVENTION 
The present invention allows the animator to create realistic character 
1 5 animation including skin and facial gestures by manipulating a detailed model of the 
actual character while at the same incorporating some of the simplifications and 
constraints of rigid body kinematics, and the ability to localize the motion that must 
be explicitly specified, to greatly reduce the number of control points whose locations 
need to be independently determined. The present invention can be viewed as a- 
20 combination of the entire kinematic approach and the dynamical approach, retaining 
the animator control of the former with the potential realism of the latter. 

In the exemplary embodiment, the skin, fur, or other covering 
(including clothing), is modeled as an elastic quasi-static surface that is a copy of, and 
elastically tied to, an underlying objected whose control points are directly controlled 
25 by me animator, either individually, or constrained as points on a plurality of 

interacting rigid bodies. The result automatically incorporates realistic stretching and 
sagging and the combined effect of multiple overlapping animation controls while 
retaining for the animator frame-by-frame control of a detailed model of the character. 

The present invention is also useful for modeling secondary features of 
30 a character, e.g., hands, for which the articulation engineer will not normally create as 
highly detailed a model as for the face. The use of the invention for modeling the 
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surface of such features provides an efficient means for obtaining realistic results with' 
a sparser (less costly) kinematic model. 

BRIEF DESCRIPTION OF THE DRAWINGS 
The file of this patent contains at least one drawing executed in color. 
5 Copies of this patent with color drawings will be provided by the Patent and 
Trademark Office upon request and payment of the necessary fee. 

FIG. 1 shows generally the elements of a computer system suitable for 
carrying out the present invention. 

FIG. 2 shows the main steps in generating a computer animation of a 
1 0 character using the techniques of the present invention. 

FIG. 3 shows the control point mesh of Geri's head. 
FIG. 4 shows some points on the skin mesh which contribute to the 
energy function. 

FIG. 5 shows Geri's head with a scalar field for k h in red. 
1 5 FIG. 6 shows a portion of Geri's hand with articulation weights set to 

the default value before smoothing. 

FIG. 7 shows Gen's hand with articulation weights as scalar field 
values derived by smoothing. 

FIG. 8 shows the result of texture mapping on a subdivision surface 
20 using the method of the present invention. 

FIG. 9 shows a neighborhood of a point in the control mesh prior to 

subdivision. 

FIG. 10 shows a region of the control mesh with a sharp crease and its 

subdivision. 

25 FIG. 1 1 shows the control mesh of Gerfs hand with sharpness 1 

features highlighted. 

FIG. 12 shows the resulting semi-sharp features in a rendered image. 
FIG 13 shows a triangular face in the initial mesh and its first 

subdivision. 
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DETAILED DESCRIPTION OF AN EXEMPLARY EMBODIMENT 

FIG. 1 shows a computer system suitable for carrying out the 
invention. A main bus 1 is connected to one or more CPU's 2 and a main memory 3. 
Also connected to the bus are a keyboard 4 and large disk memory 5. The frame 
5 buffer 6 receives output information from the main bus and sends it through another 
bus 7 to either a CRT or another peripheral which writes the image directly onto film. 
To illustrate the present invention we will describe its use in the animation of a 
character, Geri. The main steps in that process are summarized in FIG. 2. 

The first step 21 involves the production of a two dimensional control 
10 mesh corresponding to a character to be modeled and animated. In the present case, 
the mesh, shown in FIG. 3, was derived from data scanned in from a physical model 
of Geri's head. Alternatively, the mesh could be created internally through a CAD 
program without a physical model. Depending on the type of geometrical primitives 
to be used, e.g. polygons, NURBS or sub-division surfaces, the set of the control 
15 points should be selected so that the chosen primitive representation provides an 
accurate fit to the original model or design and captures all poses of the character 

during the animation. 

In an exemplary embodiment, Geri's head, depicted in FIG. 3, the 

control mesh contains approximately 4100 control points grouped to form 

20 approximately 4300 polygon faces. The points were entered manually using a touch 

probe on a physical model. Much care was taken in selecting the control points. 

Regions with fine scale details require more control points, i.e., smaller polygons. In 

addition, regions with larger polygons -- greater spaced control points - should not 

directly abut regions of small polygons, instead, to prevent artifacts in the rendered 

25 image the number and position of control points should be chosen so that transitions 

between regions of high and low detail (i.e., little and big polygons) are smooth. In 

addition, one should select the control points so that creases, or other semi-sharp 

features lie along polygon edges (the treatment of these features will be discussed in 

detail below). Control points are stored as a list which includes an identifier and the 

30 three dimensional coordinates for each point. Points are grouped as polygons and 
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stored in another list with a polygon identifier and the identifiers of the included 
points. Sharp features are also stored as ordered lists of vertices and sharpness values. 

Once the set of control points, polygons and creases defining the 
kinematic head are entered and stored in the computer, the articulation engineer must 
5 determine how each point is to move for each gesture or character movement. This 
step 22 is done by coding animation controls, which effectuate transformations of the 
model corresponding to different movements, e.g., jaw down, or left eyebrow up. The 
transformations may be either linear displacements or, as in the "jaw down" control, 
rotation about a specified axis. The magnitude of the control, i.e., the amount of 

10 displacement or angle of rotation is specified by a scalar parameter, s, supplied by the 
animator. The corresponding transformation of the model can be written as a function 
T(s) . The possible movements of Geri's face are specified using approximately 252 
such controls. The articulation engineer can specify the relative motion of different 
control points under a given animation control by assigning weights to the control 

1 5 points. In the case of Geri's head, each of the 252 animation controls affects the 

motion of approximately 150 control points to which non-zero weights are assigned. 
The effect of the weight is to specify the "strength" of the transformation at the given 
point, i.e., the transformation T with strength s at a point with weight w is given by 
T(w*s) . The weights for a given control can either be assigned individually, control 

20 point by control point, or as a smooth scalar field covering the portion of the mesh 
whose location is altered by the transformation. The scalar field approach, described 
in detail in below, offers a great improvement in efficiency over the point-by-point 
approach which, for 252 controls times an average of 150 weights per control, would 
require individually specifying almost 40,000 weights to specify the motion of Geri's 

25 head alone. 

The next step 23 in FIG. 2 is the creation of a second mesh of points, 
the skin mesh, which will typically be in one-to-one correspondence with the control 
points of the kinematic head. This need not be the case, however. In some 
embodiments, a skin mesh is used with a greater number of control points than are 
30 contained in the kinematic head and a projection of the skin points to the kinematic 
head is defined so that each skin point is "tied" to a unique point on the kinematic 
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head (which need not be a control point). Other embodiments may have a skin mesh , 
that is sparser than the underlying kinematic head. In that case, the same projection 
strategy can be used to tie skin control points to points on the kinematic head. 

In the exemplary embodiment of Geri's head, the positions of the skin 
5 points were determined by copying the kinematic head control points while Geri's 
face was in the "neutral pose", which was taken to be the expression on the model 
head when it was scanned in with the touch probe. The "neutral pose" defines the 
equilibrium positions of the skin points, i.e., the position of Geri's features for which 
his skin mesh control points will exactly match the positions of the control points of 

1 0 the kinematic head. 

In step 24, the articulation engineer specifies the properties of Geri's 
skin and the degree to which it is constrained to follow the underlying kinematic head. 
This is done by specifying the local parameters of an energy function, the extrema of 
which determine the position of the quasi-static surface, which in the present example 
15 is the skin on Geri's head. Though the details of the energy function used to simulate 
the skin on Geri's head is illustrative, variations are possible and will be readily 
apparent to those of ordinary skill in the art. The present invention is thus in no way 
limited to the details of the described implementation. That said, the energy function 
used to model Geri's head takes the 
20 form: 

t£ ' non-edge Spiels facts P° 

pairs P airs 

The first term accounts for stretching of the skin surface and is given by: 
£,1(^2)= M P > - ^P 2 1 - where P, and P 2 , as shown in FIG. 4, are the 
locations (in R 3 ) of two nearest neighbor points connected by an edge of the mesh and 
25 R is the distance between the points in the neutral pose. K sX is a spring constant which 
can either be specified globally or locally for each edge as a combination of scalar 
field values for the two points, e.g., ^,,(P„/ ) 2 ) = S .( /> .) + S t (P 2 ) where S t (P.) isa 
scalar field defined on the mesh by one of the methods described below. The second 
term includes contributions from nearest neighbor points which are not connected by 
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an edge, e.g., points across a diagonal in a quadrilateral. E s2 (P x , P 3 ) has the same 
form as the first term but with a potentially different spring constant K s2 which again 
may be constant through out the mesh or defined locally by a second scalar field, 
K s2 (P x ,^3) = $2 { p x ) + s 2 i p i ) • The P° int function 

5 E p (P x , P 2 ,P 3 ) = K p (\(P 3 -P 2 )/D X -(P 2 - P { ) ID 2 \-R p ) 2 is included to penalized 
bending and includes a contribution from each connected triplet of points. D x is the 
distance in R 3 between P 3 and P 2 in the neutral pose similarly D 2 is the distance 
between P 2 and P x in the neutral pose. R p is the value of 

|(p 3 _ p 2 ) / d x - (p 2 - i> ) / D 2 j when all three points are in their neutral pose 

10 positions (note that this is not generally zero because (P 3 - P 2 ) and (P 2 - P x ) are 
vectors in i? 3 ). The four point function, 

E d (P, ,P 2 ,P 3 ,P A ) = K d E s2 (P t , P z )E s2 (P 2 , P 4 ) includes a contribution from each 
quadrilateral face and is included to penalize skewing. The coefficient K d may either 
be a constant throughout the mesh or a combination of scalar field values at the four 
15 vertices. 

The last term in the energy function penalizes the skin points for 
straying from their corresponding kinematic mesh points: E h (P s ) = K h \P s -P k \ 
where P s is a skin mesh point and P k is its corresponding point in the kinematic 
mesh. The spring constant K h is generally given by a scalar field associated with 
20 either the dynamic skin mesh or underlying kinematic mesh. 
Defining Scalar Fields 

Scalar fields were used at several points in the above described process 
in order to define smoothly varying parameters on either the kinematic mesh or skin 
mesh. These include the articulation weights which specify the relative movement of 
25 kinematic mesh control points under the different animation control transformations, 
as well as the various spring constants and other locally variable parameters of the 
energy function. When modeling with subdivision surfaces, the control point mesh is 
a polygonal mesh of arbitrary topology, on which one cannot define global surface 
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coordinates and on which there is no "natural" or "intrinsic" smooth local 
parameterization. The absence of a local parameterization has slowed the adoption of 
subdivision surfaces as a means for modeling objects in computer animation, in part, 
because it was believed that a surface parametrization was necessary to define scalar 

5 fields and perform parametric shading, e.g., texture or surface mapping. 

One aspect of the current invention is a solution to this problem and a 
method for consistently defining scalar fields on subdivision surfaces, including scalar 
fields which can be used as "pseudo-coordinates" to perform parametric shading. 
Three exemplary methods for defining smoothly varying scalar fields on arbitrary 

1 0 polygonal meshes which can be consistently mapped through the recursive 
subdivision process are described below. 
Painting 

The first method for defining scalar fields is by painting them directly 
onto the surface mesh. In this technique, an image of a portion of the surface on 
1 5 which the field is to be defined is painted using a standard two-dimensional computer 
painting program, e.g., Amazon, the intensity of the color applied to the image is 
chosen to represent the desired magnitude of the scalar field on the corresponding 
portion of the painted surface, e.g. if one wants the skin of Geri's forehead to more 
closely follow the movement of his kinematic head than the flesh in his jowls, one 
20 would paint his forehead correspondingly darker when "applying" the field giving rise 
to the parameter k„ in the above described energy function. 

The first step in the painting method is to perform a single subdivision 
of the control point mesh in the region on which one intends to define the scalar field 
in order to obtain a mesh with only quadrilateral faces, as will always be the case after 
25 the first subdivision with Catmull-Clark rules (discussed in detail in the Subdivision 
section below). The faces in the once subdivided mesh are then numbered and 
separately coordinatized with two dimensional coordinates u and v assigned to each 
vertex (thus the need for quadrilateral faces). The surface is then further subdivided 
one or more additional times so that the resulting mesh sufficiently approximates the 
30 smooth surface limit. The u, v, coordinates for each patch (face in the once 

subdivided mesh) are carried through these additional subdivisions and coordinatize 
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new vertices within the faces of the once subdivided mesh. The image is then 
rendered and painted with a two dimensional painting program such that the 
distribution and intensity of the applied color represents the desired local magnitude 
of the particular scalar field being defined. The painted image is then scanned and 
5 patch numbers, u, v, coordinates, and color intensity are stored for each pixel. Each 
patch is then further subdivided until a dense set of points is obtained within each 
patch including points with u, v coordinates close to those stored for each pixel in the 
painted image. The scanned pixel color intensities are then assigned as target scalar 
field values to the corresponding points in this further subdivided mesh. 
10 Going back to the initial undivided mesh, one then assigns to its 

vertices initial guesses for the scalar field values which one hopes after several 
subdivisions will reproduce the target values. The mesh is then subdivided to the 
level of refinement at which the target values were assigned with scalar field values 
calculated from the initial guesses using the same subdivision rules used to calculate 
1 5 vertex locations in the refined mesh. The values of the scalar field at the target points 
resulting from subdividing the mesh with the initial guesses is then compared with the 
target values at the corresponding points. Differences from the target values are 
computed and averaged back, reversing the subdivision calculations, to find 
corrections to the initial guesses for scalar field values at the vertices in the 
20 unsubdivided mesh. This comparison and correction process is iterated until 

convergence. The result is a smooth field that closely approximates the painted 
intensities defined for any level of subdivision including the smooth surface limit. 
Smoothing 

The second more direct, if less picturesque, method for defining scalar 
25 fields is through smoothing. One begins by specifying values of the desired field 

explicitly at some boundary or known points in a region of the mesh and solving for 
the rest through an iterative relaxation approach constraining the scalar field value of 
the vertices to be equal to the average of their nearest neighbors. FIG. 5 shows GerTs 
face with the k h ("glue" field) shown in red. Smoothing was used in applying the k h 
30 field to his chin region. The use of this method to assign smoothly varying 

articulation weights is illustrated in FIGS. 6 and 1, To assign articulation weights to 
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control points in the transition region between Geri's thumb which will move with full . 
strength, w=l under the control, and his palm which will not move at all, i.e., have 
weight O under the control, one begins by enforcing these conditions and, as shown in 
FIG 4, by giving the control points in the intermediate region default values of 1 . One 
5 then performs the iterative relaxation smoothing calculation until one obtains the 

result shown in FIG. 5., a smooth interpolation of intermediate scalar field values over 
the control points in the transition region. 
F.ner gv Method 

Scalar fields can also be used as pseudo coordinates to enable 
10 parameteric shading, e.g., texture and surface mapping. One reason that subdivision 
surfaces have not be more widely adopted for modeling in computer graphics is that 
the absence of a surface parameterization was thought to prevent the use of texture 
maps. One can, however, utilizing one aspect of the present invention, define s and t 
scalar fields ("pseudo-coordinates") on regions of the subdivision surface which can 
15 serve as local coordinates for texture mapping. One cannot, of course, trick topology, 
so these pseudo coordinate fields can only be consistently defined in topologically flat 
regions of the surface. One can then patch these regions together either 
discontinuous* (e.g., if one wanted two different pieces of cloth to abut along a seem) 
or continuously (by requiring fields in overlapping regions to have common values). 
20 Pseudo-coordinates need only be defined, and consistency requirements need only be 
enforced after the model has been animated and only in regions in which the texture 
mapping is to be applied. These static pseudo-coordinate patching constraints are thus 
far easier to deal with and satisfy than the ubiquitous kinematic constraints required to 
model complex objects with NURB patches. 
25 Though s and t fields can be defined using either the painting or 

smoothing method described above, an elaboration of the smoothing method, the 
energy method, is useful for defining scalar fields to be used as pseudo-coordinates 
for texture mapping. To avoid unacceptable distortions in the surface texture, the 
mapping between the surface in R 3 and the s and t pseudo-coordinate 
30 parameterization of the texture sample in R 2 should be at least roughly isometric (i.e. 
preserve lengths and angles). If one pictures the mapping of the two dimensional 
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texture onto the surface as fitting a rubber sheet over the surface, one wants to 
minimize the amount of stretching and puckering. If the mapping is onto a portion of 
the surface with curvature, a pucker free mapping is not possible. A best fit, so to 
speak, can be achieved, however, by utilizing the first two terms in the energy 
5 function described above restricted to two dimensions. Though this technique for 
achieving a "best fit" approximate isometry is useful for texture mapping onto other 
curved surfaces, including those defined by NURBS, as will be readily apparent to 
one of ordinary skill in the art, its implementation will be described in detail for the 
case of subdivision surfaces. 
10 The aim is to insure that as closely as the geometry allows distances 

between points measured along the surface embedded in R 3 equal distances measured 
between the same points in the flat two dimensional s and t pseudo-coordinates of the 
texture patch for all points on the surface to which the texture mapping is to be 
applied. To achieve a good approximation to this best compromise solution, we begin 
15 by subdividing the initial mesh several times in the region to be texture mapped. This 
provides a fine enough mesh so that the distances between nearest neighbor vertices 
measured in /J 3 sufficiently approximate the distance between the same points 
measured along the surface. One then generally orients the texture on the surfaceby 
assigning s and t pseudo coordinates to several control points. One then completes the 
20 mapping, finding pseudo-coordinate s and t values for the remaining mesh vertices, by 
minimizing a two dimensional energy function over the region of the mesh on which 
the texture is to be applied. An exemplary energy function contains the two point 
functions from the energy function defined above, i.e., 

edge non-edge 
pairs pairs 

25 with as before E t (P l9 P 2 ) = k,(\P x - P 2 |- R) 2 butnowRisthe 

distance between the points as measured along the surface (as approximated by adding 
distances in three space along the subdivided mesh) and \P X - P 2 \ is the two 
dimensional Euclidean distance between the points' s and t pseudo-coordinates, i.e., 

-5 2 ) 2 +(/, -* 2 ) 2 • T 11 * 8 ener gy function can be minimized using Newton's 
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method or other standard numerical technique as described below for minimizing the * 
full energy function to determine the position of skin points. 



5 finished. The remaining steps can all be performed "off-line" as a precursor to the 
actual rendering of the scene. Because these steps do not require additional input by 
the animator and take a small fraction of the computation time required for rendering, 
they add little to the time required for the overall animation project. 

Minimizing the Energy Function to Determine the Position of Skin Points 

1 0 The positions of the control points for the surface mesh are determined 

by minimizing the energy function. This can be done iteratively using Newton's 
method or other standard numerical technique. For each frame of animation, the 
location of the skin control points can be taken initially to correspond with the 
positions of the control points of the kinematic mesh. A quadratic approximation to 

15 the energy function is then computed and minimized yielding approximate positions 
for the control points for the surface mesh. These approximate positions are used in 
the next quadratic approximation which is solved yielding more refined approximate 
positions. In practice, a few iterations generally suffice. 

To drive the system to a definite solution in the situation in which the 

20 energy function has a null space, an additional term is added to the energy function for 
each point in the dynamic mesh which, for the nth iteration, takes the form 



iterations. Though the discussion of the exemplary embodiment to this point has dealt 
25 primarily with the quasi-static determination of the surface control point locations, 
this degeneracy breaking term can be slightly modified to allow for the incorporation 
of dynamic effects as well, e.g., momentum of the surface mesh. To do so one can 



One can then treat the iteration index as indicating time steps as well and update the 



Off-Line Steps 

At this point in the process all of the manual steps in the animation are 




E m =aN\p;' -2Pr' +P, 



2 



replace the above described term with one of the form 
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underlying kinematic control point positions in steps between frames. The 
incorporation of momentum in this manner may be useful in some cases, e.g., to 
automatically incorporate jiggling of fleshy characters. 
Subdivision 

5 Once the positions of the skin mesh vertices are determined by 

iteratively minimizing the energy function, the model can be refined prior to rendering 
in order to produce a smoother image. One method for refining the mesh is through 
recursive subdivision following the method of Catmull and Clark. See E. Catmull and 
J. Clark. Recursively generated B-Spline surfaces on arbitrary topological meshes. 

10 Computer Aided Design, 10(6):350-355, 1978. One could apply other subdivision 
algorithms, e.g., Doo and Sabin, see D. Doo and M. Sabin. Behavior of recursive 
division surfaces near extraordinary points. Computer Aided Design, 10(6): 356-360, 
1978, or Loop if one chose the initial mesh to be triangular rather than quadrilateral, 
see Charles T. Loop. Smooth subdivision surfaces based on triangles. M.S. Thesis, 

1 5 Department of Mathematics, University of Utah, August 1 987. In the current example 
Catmull-Clark subdivision was used in part because a quadrilateral mesh better fits the 
symmetries of the objects being modeled. In the limit of infinite recursion, Catmull- 
Clark subdivision leads to smooth surfaces which retain the topology of the initial 
mesh and which are locally equivalent to B-Splines except in the neighborhoods of a 

20 finite number of exceptional points at which the initial mesh had a vertex or face with 

other than 4 edges. 

The details of constructing smooth surfaces (and approximations to 
smooth surfaces) through subdivision is now well known in the art and will only be 
discussed briefly for completeness and as background to the discussion of hybrid 

25 subdivision below. FIG. 9 shows a region about a vertex in a quadrilateral mesh. 
This may be either the initial skin mesh or the mesh resulting from one or more 
subdivisions. The vertex has n edges. For a "normal point," n=4. If n is not equal to 
4, the point is one of a finite number of exceptional points which will exist in any 
mesh that is not topological^ flat. The number of such points will not increase with 

30 subdivision. FIG. 10 shows the same region overlaid with the mesh resulting from a 
single subdivision. The positions of the resulting vertices which can be characterized 
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as either vertex, edge, or face vertices, are linear combinations of the neighboring 
vertices in the unsubdivided mesh: 

1=1 

Where j runs from 1 to n, the number of edges leaving the vertex S. 
Capital letters indicate points in the original mesh and small letters represent points in 
the next subdivision. The subdivision process can be written as a linear 
transformation taking the vector V = (S,R l9 ... 9 R m9 Q t , ...,Q„) T to the vector 

v = (s,r t9 ... 9 r n9 q l9 ...,q ll ) T defined by the 2n+l by2n+l square matrix M n : 

v = M n V . For Catmull-Clark subdivision about an ordinary point, i.e., when n=4, 

using the subdivision formulas given above M n takes the form: 

'» Yi K % % % Y 1 



M 4 = 1 / 16 
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In practice, the subdivision procedure is applied a finite number of 
times, depending on the level of detail required of the object which may be specified 
explicitly or determined automatically as a function of the screen area of the object in 
a given scene. Typically if one starts with an initial mesh with detail on the order of 
the control mesh for Geri's head, 4 or 5 subdivisions will suffice to produce 
quadrilateral faces corresponding to a screen area in the rendered image of a small 
fraction of a pixel. Iterating the above formulas gives the positions and scalar field 
values for the vertices in the finer subdivided mesh as a function of the positions and 
scalar field values of the control mesh. 
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After a sufficient number of iterations to produce enough vertices and - 
small enough faces for the desired resolution, the vertex points and their scalar field 
values can be pushed to the infinite iteration smooth surface limit. In the case of 
Catmull-Clark surfaces, the smooth surface limit is a cubic spline everywhere but in 
5 the neighborhoods of the finite number of extraordinary points in the first subdivision 
of the original mesh. These limit values are easily determined by analyzing the 
eigenspectrum and eigenvectors of the subdivision matrix, M„. See M. Halstead, M. 
Kass, and T. DeRose. Efficient, fair interpolation using Catmull-Clark surfaces, 
Computer Graphics (SIGGRAPH 1993 Proceedings), volume 27, pages 35-44, 
10 August 1993. Specifically, applying the matrix M n to a vector an infinite number of 
times projects out the portion of the initial vector along the direction of the dominant 
right eigenvector of the matrix M„ . If we take m = 2n + 1 , the size of the square 
matrix M n , then we can write the eigenvalues of M n as e x > e 2 >....e OT with 
corresponding right eigenvectors We can then uniquely decompose V as 

15 V = c x E x + c 2 E 2 +.„c m E m 

Where the Cj are three dimensional coordinates and/or values of 
various scalar fields on the surface but are scalars in the vector space in which M n 
acts. Applying M n we get M n V = e x c x E x + e 2 c 2 E 2 +...e m c m E m . For (M„) c °Fto 
have a nontrivial limit the largest eigenvalue of M„ must equal 1 . So that 
20 (M n )" V = c x E x . Finally, Affine invariance requires that the rows of M n sum to 1 
which means that E x = (l,...,l). Which gives s" =c x . 

If one chooses a basis for the left eigenvectors of M n , L x , . . . L m so that 
they form an orthonormal set with their right counterparts, i.e., L } • E k = 8 Jk , this 
projection is given by the dot product L x • V where L x is the left eigenvector of M n 
25 with eigenvalue 1 and V is the column vector defined above of points in the 

neighborhood of S. For Catmull-Clark subdivision, the value of the this dot product 
and thus the position of the point s after infinite subdivision is given by 

= 1/, An 2 s + 4Y r.+V^-L This formula gives not only the smooth 
/n(n + 5)\ ' j-{ ') 
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surface limit value of the position of the vertex points in /? 3 but can also be used to 
calculate the smooth surface limit of any scalar fields at those points as a function of 
their values at points in the neighborhood of S. 

Similarly, with a bit more math, see Halstead, et al. cited above, it can 
be shown that the eigenvectors for the second and third largest eigenvalues of the 
local subdivision matrix M n span the tangent space of the limit surface at the point 
s* . One can again project these out of the vector V using the orthonormality 
property of the left and right eigenvectors giving c 2 = L 2 V and c 3 = L 3 " • V . 

Because c 2 and c 3 span the tangent space at the point s™ , their cross product gives a 

vector normal to the limit surface at that point, i.e., N" 3 = c 2 x c 3 at the point . 
These tangent vectors have been calculated for Catmull-Clark subdivision and are 
given by: 

c 2 = A n cos(2m I n)r f + (cos(2;ri / n) + cos(2^(i + l) / n))^ 

Where A„ = 1 + cos(2tt / n) + cos(^r / «)V 2 ( 9 + cos(2^ / ri) 
and c 3 is obtained by replacing r, with r l+I and q t with q M . 

After constructing AT" , the smooth surface limit normal, at each of the 
vertices of the refined mesh. One can then interpolate these normals across the 
subpixel faces of the refined mesh to render the character using Phong shading. 

Hybrid Subdivision Schemes 

One aspect of the present invention is a method for creating surfaces by 
sequentially combining different subdivision rules. By subdividing a mesh a finite 
number of times with one or more sets of "special rules" before taking the infinite 
subdivision limit with the "standard rules", one can produce different limit surfaces 
from the same initial mesh. Combining different subdivision rules thus provides an 
additional degree of freedom which can be a more efficient means for obtaining the 
desired limit surface than adding points to the initial mesh. Below we describe in 
detail two particular exemplary uses of this aspect of the invention in modeling Geri. 
Many others will be readily apparent to those of ordinary skill in the art. 
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Semi-Sharp Edges and Creases 

Human skin is not a completely smooth surface. Faces and hands, for 
example, contain somewhat sharp features, both creases and edges. To model these 
features using the general smooth-surface-iimit subdivision techniques outlined 
5 above, or more conventional B-spline patch methods, would require a very 

complicated initial control mesh in which the vertex spacing in regions of sharp 
features would have to be very small, leading to lots of points and quadrilaterals. One 
method for modeling sharp features with subdivision surfaces without adding 
substantially to the complexity of the initial control mesh is to alter the subdivision 
10 rules for points lying on corners or sharp edges or creases. As described above, these 
features can be incorporated and tagged in the initial control mesh. 

The sharpness of these features can be maintained throughout the 
otherwise smoothing subdivision process by decoupling the subdivision process along 
sharp edges so that points on either side of the sharp feature do not contribute to the 
1 5 position of points on the edge in subsequent subdivisions. Locally modifying the 

subdivision algorithm in this way results in a limit surface with sharp edges or creases 
across which the tangent plane is discontinuous. Details of this method for modeling 
various sharp features using Loop subdivision on a triangular mesh can be found in 
Hoppe, et al. Piecewise smooth surface reconstruction. Computer Graphics 
20 (SIGGRAPH '94 Proceedings) (1 994) 295-302. 

The problem with using the Hoppe, et al. approach for the realistic 
animation of characters is that skin surfaces do not have infinitely sharp edges. All 
real objects, including humans, have some finite radius of curvature along their 
otherwise sharp features. This fact may be ignored when rendering machined metal 
25 objects but is very apparent when modeling and rendering human faces. The 

exemplary embodiment includes a modification of the above described subdivision 
algorithm to allow the modeling and rendering of edges and creases of arbitrary and 
continuously variable sharpness. 

FIG. 1 1 shows a portion of the initial control mesh of Geri's hand with 
30 sharp features highlighted. As described above, care was taken in the initial choice of 
mesh vertices so that these sharp features lay along edges of the initial mesh. Their 
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position in the mesh was stored along with a sharpness value between 0 and 5 
(assuming 5 explicit subdivisions was the maximum performed before pushing the 
vertices to their limits). Because the skin mesh has the same topology as the control 
mesh, the corresponding edge in the skin mesh is uniquely identified. To model an 

5 edge or crease with sharpness N where N is an integer, one applies the sharp edge 
subdivision technique described in Hoppe et al. to determine the locations of points 
along the designated edge or crease for the first N subdivisions. Thus to determine 
new edge and vertex locations along the crease or edge instead of applying the 
standard Catmull-Clark formulas described above, one uses for example the following 

1 0 sharp edge formulas: 

s^j^R. +y%R k +y$ s where Rj and R k are the points on either 

side of S along the sharp edge or crease, as shown in FIG. 10. If S is a vertex at 
which a crease or edge ceases to be sharp its position after subdivision is calculated 

1 5 using the normal Catmull-Clark formulas. 

After N subdivisions one applies the normal smooth algorithm for 
subsequent explicit subdivisions and then pushes the points to their smooth surface 
limits. Thus for sharpness 2 features, one applies the sharp formulas for the first two 
subdivisions and then uses the smooth formulas for subsequent subdivision and for 

20 taking the smooth surface limit. One can get "fractional sharpness" e.g., 1.5, by 

linearly interpolating edge point locations between the sharpness 1 and 2 results, i.e., 
calculate the position of control points after performing one sharp and one smooth 
subdivision and calculate the position of control points after subdividing twice with 
the sharp subdivision rules and take the average of the two locations before 

25 subdividing farther with the smooth rules or pushing the points to their smooth surface 
limits. 

One can also determine the positions and tangents on the limit surface 
of points in the initial mesh on a semisharp feature without explicitly iterating the 
semisharp rules, by analyzing the eigenstructure of the combined transformation 
30 matrix in a manner similar to that described above for the case of smooth subdivision. 
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The limit surface position for the neighborhood of a semisharp feature of sharpness k - 
is given by 

= (A/ OTOO/A y (M^y V where we have suppressed the n index 
indicating the number of edges and added a subscript designating smooth and sharp 
5 transformations. As above for the smooth case, we can replace the application of 
M smooth ^ infinite number or times with dotting by L x its dominant left eigenvector, 
giving v* = L x • (M^^Y V . We can then replace L x by its decomposition in terms 
of left eigenvectors of M sharp . L x =t x l x + t 2 l 2 +..././„ where /, are left eigenvectors 
of M sharp and f, are expansion coefficients, scalars under . Applying k 

10 times from the right to this expansion, we get: v" = (t x X x k l x +...t m X m k l m ) wherethe 
Xj are the eigenvalues of M Aarp . 

One may also create a feature with varying sharpness along its length. 
Assuming again that the initial mesh is created so that the sharp feature lies along one 
or more connected edges, one can specify sharpness values at the vertices of the 
1 5 feature either by hand or by one of the scalar field techniques described above, limited 
to one dimension. One can calculate the limit values of the sharpness field treating 
the sharp feature as a subdivision curve and taking its limit value using the dominant 
left eigenvector as described above for two dimensional surface subdivision. For 
uniform cubic B-Spline subdivision along a curve, that vector is given by 
20 u x = 1 / 6[l,4,l] . See, e.g, E. Stollnitz, T. DeRose and D. Salesin. Wavelets for 

Computer Graphics, 1996, 61-72. The limit value of the field on a point along the 
semi-sharp feature is thus given by 2/3 of its value plus one 1/6 of the value of each of 
its nearest neighbors. 

The location of each vertex point on the semisharp feature is then 
25 calculated using the limit value of its sharpness field, i.e., one applies the sharp rules 
in the neighborhood of that point the number of times specified by the point's 
sharpness value (including interpolation, as described above for non-integer sharpness 
values). 
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This continuous smoothing of sharp edges can be used in conjunction - 
with other subdivision schemes as well including those proposed by Loop and Doo 
and Sabin, cited above. In particular, all of the sharp edge techniques and results of 
Hoppe, et al. can be extended to allow for the efficient modeling of semi-sharp 
5 features by combining smooth and sharp subdivision rules in the manner described 
above for Catmull-Clark subdivision. 

Improving The Surface In the Neighborhood of Triangular Faces 
A second application of hybrid subdivision used in the exemplary 
embodiment involves a modification to the standard Catmull-Clark rules used to 
1 0 determine the location of face points in the first subdivision of a triangular face. As 
described above, if one begins with an arbitrary polygon mesh which includes non- 
quadrilateral faces, after one subdivision with Catmull-Clark rules, all faces in the 
refined mesh will be quadrilateral (though some vertices will have more than four 
edges). If one has an initial mesh which is convex with some triangular faces, e.g., 
15 the initial control point mesh of Geri's head, the smooth surface resulting from 

standard Catmull-Clark subdivision will be somewhat "lumpy" in regions in which 
there were triangular faces in the original mesh. The shape of the limit surface can be 
improved if when calculating the location of the face point for a triangular face in the 
first subdivision (the only time there will be triangular faces), one takes its location to 
20 be the average of the locations of its surrounding edge points after subdivision rather 
than the average of the its surrounding vertices in the undivided mesh, the standard 
Catmull-Clark rule. This is illustrated in FIG. 13. The solid lines and points P t are 
points in the initial mesh. The dashed lines show the once subdivided mesh with 
points c, and / . Under normal Catmull-Clark rules the location of / is given 
25 by:/ = 1 / 3(/> + P 2 + P 3 ) . Instead, we take / = 1 / 3(e, + e 2 + c 3 ) . Because the e, 
depend the locations of neighboring faces as well as the /> , , / will be drawn closer to 
the center of a convex mesh and the limit surface will have a smoother contour. 

The specific arrangements and methods described herein are merely 
illustrative of the principles of the present invention. Numerous modifications in form 
30 and detail may be made by those of ordinary skill in the art without departing from the 
scope of the present invention. Although this invention has been shown in relation to 



WO 99/06962 PCT/US98/15702 

23 

particular embodiments, it should not be considered so limited. Rather, the present 
invention is limited only by the scope of the appended claims. 
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CLAIMS 

What is claimed is: 

1 1 . A method for enhancing the appearance of characters in computer 

2 animation comprising the steps of: 

3 a. constructing an articulated model of the character which can be 

4 kinematically controlled by the animator; 

5 b. constructing a quasi-static surface coupled to said articulated 

6 model, wherein the position of said quasi-static surface depends on the position of 

7 said articulated model. 



2. The method of claim 1 wherein the position of said quasi-static surface 
corresponds to that of said articulated model for a particular configuration of said 
1 0 articulated model. 

1 3. The method of claim 1 wherein said quasi-static surface is elastically 

2 coupled to said articulated model. 

1 4. The method of claim 3 wherein the strength of said elastic coupling can 

2 be locally varied throughout said articulated model. 

1 5. The method of claim 1 wherein the position of the quasi-static surface 

2 depends on one or more internal parameters of the quasi-static surface. 



1 6. The method of claim 5 wherein said one or more internal parameters 

2 include the flexibility of the surface. 

1 7. The method of claim 5 wherein said one or more internal parameters 

2 include the elasticity of the surface. 

1 8. The method of claim 1 wherein the position of said quasi-static surface 

2 is determined by the extrema of an energy function. 
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